Universal Source Coding

The material covered in this thesis falls mainly in two parts. In the first part, which is found in chapter II, the foundations of source coding are considered. A careful introduction to the notion of codes for symbols, blocks and infinite sequences is given. The classical notion of a stationary ergodic source is then introduced as a natural concept arising when assuming limiting statistical consistency of infinite sequences. Using the recently developed techniques of packing and counting, a proof of the entropy-rate theorem (the Shannon-McMillan-Brieman theorem) is given, which ultimately leads to an almost sure version of Shannons coding theorem due to Barron. As a parallel to this, a result of Ryabko, which might be thought of as a coding theorem for combinatorial (non-probabilistic) sources, is presented. The first part of the thesis closes with a brief overview relating the theoretical bounds to codes that are practically implementable on a computer. In the second part, which is found in chapters III and IV, the framework developed in the first part is applied to the well known and widely used Lempel-Ziv codes. Using the 1978 variation as prime example, a description of the algorithm is given, and a proof that the corresponding code is weakly universal on the class of stationary ergodic sources is made. This proof is based, once again, on the packing and counting techniques forwarded by Ornstein, Weiss and Shields. In order to gain better insight into the Lempel-Ziv code a symbol-wise analysis of the code is conducted and a recently developed proof (due to Yang and Kieffer) of the second order performance of the Lempel-Ziv code when applied to an i.i.d. source is given. Finally, a survey of recent results from the non-probabilistic domain is presented. It covers the challenges of individual coding, and the ideas of Shields, Ryabko and Suzuki regarding the use of individual sequences and Hausdorff dimension as a new method for comparing universal codes.